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INTRODUCTION 

During the last twenty years, researchers at Ames have been investigating the design of pictorial 
displays for civilian aerospace applications. In the mid-1980's this work sparked public imagination 
with the suggestion that the experience of a virtual environment provided by a head-mounted aircraft 
simulator could be adapted inexpensively for a wide variety of other applications. It was suggested that 
these head-mounted devices, called by Ivan Sutherland the "ultimate computer display," could become 
the ultimate computer interface. They were thought to be able to take computer users beyond the two- 
dimensional desktop metaphor for computer operation into a three-dimensional environment metaphor. 
They were billed as the "disappearing" interface, but as is currently clear, they are still far from 
"disappeared" (Ref. 1). 

One may get a sense of the background of some of this work at Ames by very briefly reviewing 
its development through a few examples of studies of human performance using interactive graphical 
displays. The essence of this work is human interface design. The characteristic innovation is that the 
interface itself appears as an environment. The goal has been to use this new format to advance the 
safety and efficiency of human-machine interfaces in commercial aircraft and spacecraft, and such 
applications have materialized (Refs. 2 and 3); but other applications can extend far beyond those in 
aerospace. 

In the course of this work the researchers at Ames have followed the long standing interest 
people have had in pictures as a communication medium. Pictorial representations have both symbolic 
and geometric components. The Lascaux Cave art, probably the oldest pictures known, emphasizes the 
original, probably primal interest in symbolism. But pictures also make more quantitative points. 

Maps are a form of pictorial representation in which the geometry is central since the message is more 
numerical, but we can see that even mapmakers cannot resist decorating their creations with symbols. 

As the graphics capability of portable computers advanced, it became possible in the 1980's to 
consider pictorial display formats for electronic displays in aircraft and spacecraft. At that time 
investigations began of the design of aircraft traffic displays incorporating perspective projections. 

The intuitive and immediate comprehensibility of such displays made them attractive for 
situations requiring quick interpretation; but study of them rapidly shows that effective communication 
requires much more than visual impact and, in particular, that the well known ambiguities of 
perspective images have to be counteracted (Ref. 4). 

In fact, research and implementation of reference systems at Ames has determined that 
considerable geometric and symbolic enhancement was necessary to make a perspective-based picture 
useful as a display for guidance and control (Refs. 5 and 6). The aircraft display shown in Fig. 1 
(lower right) is not just a pretty picture. Some of the enhancements that transform it from what one 
could call a spatial display into a spatial instrument, a display designed for a purpose, are described in 
the caption. 
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Figure 1 - Simply rendering an aircraft traffic pattern in perspective relative to a reference aircraft 
(center of display) does not in itself provide much spatial sense of the layout (upper left). Adding a 
reference gnd helps but relative altitude is still hard to perceive (upper right). Adding drop-reference 
lines and coaltitude x s to show the reference aircraft’s altitude against all others helps, but aircraft 
aspect is still ambiguous (lower left). Adding predictor lines with additional drop reference lines helps 
remove aspect ambiguity and resolve future traffic conflicts (lower right). 

See Refs. 5 and 6 for more details. 







Thus, as drawn by our E&S Picture System this display had many physically unrealizable 
characteristics, called geometric enhancements. These enhancements turned the display into an 
instrument. They allowed pilots to easily use the display to correct flight path errors, despite 
difficulties associated with rendering. Perspective ambiguities were only one such problem considered. 
But another way to avoid some of the uncertainties associated with perspective images is to jump into 
the picture itself. 




Figure 2 - Visual virtual environment display systems have three basic parts: a head-referenced visual 
display, head and/or body position sensors, a technique for controlling the visual display based on head 
and/or body movement. One of the earliest systems of this sort developed by Philco engineers (Ref. 7) 
used a head-mounted, binocular, virtual image viewing system, a Helmholtz coil electromagnetic head 
orientation sensor, and a remote TV camera slaved to head orientation to provide the visual image. 
Today this would be called a telepresence viewing system (upper left panel). It was used to control a 
remote camera (upper right panel) that was controlled by the user's head position. The first system to 
replace the video signal with a totally synthetic image produced through computer graphics was 
demonstrated by Ivan Sutherland for very simple geometric forms (lower panels) (Ref. 8). 


The first implementation of a display system that allowed a viewer to do this is illustrated here 
in a figure from the early 1960's (Fig. 2). It was built by two Philco engineers, Brian and Comeau 


(Ref. 7). 
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This device illustrates the now familiar components of all subsequent systems using head- 
referenced displays. Sensors to monitor body positions, i.e., head and hand position trackers, 
effectors to present visual information, i.e., a miniature video display with accommodative relief, and 
some interlinkage hardware to connect the other two components. 

These components combine in an active control loop which gives users the impression that they 
are located at a remote work site (Fig. 3). 



Figure 3 - Information flows in a virtual environment simulation or telepresence display. 


In the case of a telepresence display the interlinkage hardware includes cameras and 
manipulators; this work site is real, but it could well be synthetic if the interlinkage hardware were a 
computer simulation. 

Thus, a central aspect of head-referenced display systems is that they amount to personal 
simulators. Often they are simulators that are worn rather than entered. Accordingly, among the first 
operational display systems of this sort were indeed, head-mounted aircraft simulators (see Fig. 4). 

Unlike traditional simulators associated with vehicular activity, the wearable systems are 
probably more uniquely suited for the simulation of manipulative activity like that associated with 
teleoperation. In particular, we have been interested in the study of the design of direction and distance 
in head-mounted displays of nearby objects; that is, objects within arms length. This application 
domain is relatively new since most head-mounted displays heretofore used in aerospace applications 
present virtual image targets at least several meters away from the user. 
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As an example of current design issues, we have studied the benefits of incorporating head-roll 
tracking for accurate presentation of direction and orientation information in a virtual environment. The 
need to include roll tracking is interesting since we could achieve shorter transmission and processing 
lags if it is left out. 



Figure 4 - Though very expensive, the CAE Fiber Optic Helmet Mounted display, FOHUD (upper 
panel), is one of the highest-performance virtual environment systems used as a head-mounted aircraft 
simulator display. It can present an overall visual field 162° x 83.5° with 5-arcmin resolution with a 
high resolution inset of 24° x 1 8° of 1 .5 arcmin resolution. It has a bright display, 30 Foot-Lambert, 
and a fast, optical head-tracker, 60-Hz. sampling, with accelerometer augmentation. 

In our experiment we slaved a stereo camera platform having three degrees of rotational 
freedom to a head-mounted display. We studied the user's ability to position and orient objects in the 
control site to corresponding objects in the remote site (Fig. 5). Despite assertions of the benefits of 
matching human head kinematics through roll tracking, pitch and yaw were enough. We found that 
disabling roll had very little effect on performance provided subjects were not required to make very 
large rotations of their head with respect to their torso (Ref. 9). 

In another example, we have studied perceptual issues that effect the apparent depth of nearby 
virtual objects presented via head-mounted displays called electronic haploscopes. They allow flexible 
control of the optical characteristics of binocular stimuli that may be optically superimposed in the 
subject's view. With this display a computer graphics image of a virtual object can be precisely 
calibrated so that factors that affect its apparent depth with respect to its intended depth may be studied. 
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Figure 5 - The upper panel shows a three dof camera platform which is slued to the head orientation of 
a head-mounted display at the control site (lower panel). Studies of the improvement of the users' 
abilities to position and orient objects at the control site to match objects at the remote site have shown 
that the addition of camera roll only improves performance if large head rotations relative to the torso 

are forced. Pitch and yaw are usually sufficient. 
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Figure 6 - An example of a head-mounted see-through display with adjustable viewing optics that 
allows independent variation in ocular vergence and accommodative demand making it an electronic 

haploscope. 


Most recently, study has focused on an interesting interaction between the apparent distance to 
the virtual object and real surfaces over which it is superimposed. Briefly, we find that after graphics 
calibration and alignment, the apparent distance of the virtual object, as indicated by adjustment of a 
real cursor, is reduced by superposition on a real surface at the apparent distance of the object. This 
phenomena has clear implications for placement of virtual objects in displays for medical visualization, 
teleoperation and mechanical assembly. Current work is directed to describe it and to understand its 
cause (Ref. 10). 

As part of interest in the dynamic aspects of the simulation loop, studies have been done on the 
intrinsic response latency of sensors and future studies on the role of latency, and update rates are 
planned for a number of perceptual and manual control phenomena, including the apparent offset of 
depth just mentioned. 

In conclusion, it is useful to recall an observation about human interaction with pictorial 
displays developed from our research on the design of aerospace instruments. The phenom ena most 
useful for the design of efficient pictorial displays are not invariably associate d with the realistic 
rendering of the images, but rather the display and control of error . In some respects a sense of 
presence is not the central communication of our images. Display images, like maps, are schematic 
pictures with a point, often a numerical point. The principle challenge of designing them is often to 
determine how to portray error in a quickly interpretable manner so corrective action can be taken. 



Alignment 

and 

Calibration 


Cursor 
Adjustment 
to Distance of 
Virtual Image 



Study Effects 
of Occlusion 
and Motion on 
Judged Position 




3 



Figure 7 - an experimental scheme for examining the effects of the judged distance to computer- 
generated rotating virtual objects after they have been superimposed on rotating physical surfaces. 

After adjustment of the display parameters to correctly display visual direction for both the left and right 
eyes independently (top panel), the depth of a stereoscopically presented virtual object can be pointed 
out with a physical cursor to an accuracy of several millimeters (middle panel). Interposition of a 
physical surface at the judged depth of the virtual object causes the virtual object to appear to move 
closer to the observer (lower panel) (see Ref. 10 for experimental details). 


Displays such as the one illustrated in Fig. 8, used to plan orbital maneuvers, do not depend 
upon photo realistic naturalness, but fluent, low-order control of animated graphical icons displaying 
constraints, status, and the future conditions of objects. In this case, a carefully controlled presentation 
of the counter-intuitive aspects of orbital mechanics was the key innovation, not making the astronauts 
feel that they were in orbit (Ref. 1 1). 

This display of the counter-intuitive and nonlinear aspects of orbital maneuvering has provided 
a basis for the design and testing of a special format that allows in situ design of minimum-fuel, multi- 
bum maneuvers in a simulated multi-spacecraft planning environment. By focusing on the symbolic, 
geometric, and dynamic enhancements to synthetic environments, it provides examples of features that 
may be usefully incorporated into actual orbital display systems. 
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Figure 8 - A display for planning short duration orbital maneuvers subject 
to realistic maneuvering constraints. 


The difficulty in informal planning of orbital missions arises from 1) the higher-order, nonlinear 
control dynamics of orbital maneuvering, 2) the counter-intuitive character of relative orbital motion, 
and 3) the frequent absence of stable reference points. The difficulties these characteristics pose for 
missions conforming to operational constraints on relative velocity, thruster plumes, and collision risk 
may be substantially overcome by visualizing the orbital trajectories and constraints in a pictorial, 
perspective display. 

Though the visualization used in the display most directly assists planning by providing visual 
feedback to aid visualization of the trajectories and constraints, its most significant novel design 
features include 1) an inverse dynamics algorithm that reduces the order of control while also removing 
control nonlinearities expected from the operator, and 2) a trajectory planning mode that creates, 
through a geometric spread-sheet, the illusion of an inertially stable environment. Consequently, the 
display is not just a "pretty picture," illustrating that computer graphics can be used to model orbital 
motion. It is rather a spatial instrument for interacting with orbital dynamics and now is in use in a 
number of laboratories around the world. 
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This synthetic planning environmental provides the user with control of relevant static and 
dynamic properties of mid-course thrusts during small orbital changes allowing independent solutions 
to the normally coupled problems of orbital maneuvering. 

This display illustrates how a synthetic environment may be defined so as to couple human 
problem solving abilities with the computer's computational capacities so as to enable interactive 
optimization of complex evaluation functions. Furthermore, it illustrates that the synthetic 
environments defined to enhance man-machine communication can benefit from informative symbolic, 
geometric, and dynamic enhancements. 

As a display of error, i.e., constraint violation, this maneuvering display is distinctly non- 
McLuhan. This media is not the message. The focus of interest is the display and control of error, not 
visual impact. Thus, the paramount consideration in the design of a virtual environment as an 
environmental instrument is the information to be communicated and the error to be controlled. 
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